Repository navigation
feat: add scheduler backend - #1282
Merged
Merged
Conversation
Preserve in-progress storage, scheduling, claims, credentials and native API/CLI work. The orchestration backend remains incomplete and is not ready for delivery.
Add claim-scoped execution inputs and native snapshot drift validation. Incomplete orchestration backend checkpoint; worker launch and the remaining control-plane integration are not implemented yet. Targeted SQLite/PostgreSQL tests pass; full static readiness is pending.
Verify pinned inputs before start and expose retryable run reports and cancellation heartbeats over the authenticated API. Disable redirects for credentialed requests and cover the lifecycle against SQLite and PostgreSQL. Keep the orchestration PR explicitly incomplete pending actual launchers and the remaining backend scope.
Separate lease expiry, replica connections, terminal run fixtures and fault cases so each integration binary imports only the support it uses. Preserve SQLite and PostgreSQL transaction coverage.
Observe heartbeat loss in fenced, bounded transactions while retaining physical write claims. Recover through existing worker heartbeats and continue full batches without an idle delay.
Own worker heartbeats on a dedicated native runtime and connection pool. Propagate cancellation and revoked credentials without cancelling work on transient API failures; stop and join with the execution owner.
Move bounded client construction, credential redaction, URL validation and JSON response handling to the API owner so worker and location flows can share them. Keep independent heartbeat pools and worker-facing error categories.
Add privileged native administration with JSON results, bounded location credentials and atomic security audit. Detect lost result delivery without claiming committed mutations rolled back.
Keep the public Python API limited to authoring records and native delegation. Resolve selectors and validate schedule data in Rust before publication. Preserve stable shorthand trigger names across cron edits. Add the explicitly approved narrow Python facade boundary entries without increasing movable Python or native crossing budgets.
Collect exported jobs and nested factory containers natively, deduplicating aliases and terminating cyclic containers. Explicit empty declarations skip user module execution. Remove failed imports from the module cache so publication retries cannot reuse partial definitions.
Verify retained compiler baselines before worker startup and join native heartbeat work on interpreter-side close. Preserve claim identity and report retry keys without treating resource teardown as physical warehouse completion.
Restrict forward and reverse cron expansion and pure tick planning to the scheduler. Preserve scheduling bodies and all existing fault, DST and simulation coverage.
Expose owner-only cancellation with stable retry identity. Commit lifecycle facts, projections and security audit together while retaining active physical claims until termination.
Resolve manual requests against the current published job inside one write transaction. Preserve the accepted snapshot and execution inputs across retries after redeployment, and commit queue publication with the trusted owner audit.
Compile in the project interpreter and retain native snapshots through publication. Keep JSON results separate from progress and distinguish uncertain acknowledgements from failed result delivery.
The decision rules were a trait with one real implementation and four broken ones that only the simulation test used. The shipped rules are now plain code. The simulation still proves its oracle finds misses by writing consumption without the started row.
Appends now go in fixed-size statements: keeping an execution inside one statement bought nothing, because separate statements are not atomic together and the decision already treats a partly written execution as unfinished. Source freshness and consumption share one state-table naming helper, and seeds use the same forced-run reasons as models. The consumption connection's close joins the allowlist of context objects that each release their own resource.
…e place Each SQL Server state index repeated the sys.indexes guard, and each keyed state table repeated the NVARCHAR(450) key narrowing. Postgres repeated CREATE INDEX IF NOT EXISTS per table. One private builder per adapter now takes the table, index and columns, and the fingerprint, freshness, node-result, audit-result and consumption methods use it. The rendered SQL is unchanged.
The process-ownership crate follows the workspace's fixed 0.0.0 crate version, like every other unreleased crate.
The Windows recovery job failed six orchestration tests with Fenced. Run serially on the same runner, five of them passed: four store-heavy tests at once on the four-core runner starved each other past the 30-second scheduler lease. The job now runs two at a time there. The sixth failed serially as well. The tick-retention test inserts a backlog of a thousand ticks one at a time after taking the lease, and on that runner the inserts outlasted it. The scheduler renews its lease between passes, and the test now renews before pruning in the same way. The temporary latency and serial-timing diagnostics are removed, and so is the CLI step whose filter no longer matched any test.
An OIDC server can now map identity-provider groups to roles. groups_claim names the ID-token claim that lists a person's groups, and each group_roles entry maps one group to a built-in or custom role, optionally limited to one location and target. A mapping cannot grant admin, and a custom role or a target needs a location. Groups are read only from the verified ID token, never from userinfo. A claim that is not a list of nonempty strings, including an overage pointer, gives no groups. People still need an explicit identity binding; groups never create accounts. Each sign-in replaces the person's recorded groups, and the server publishes its mapping at startup, so authorization joins the two live: removing a mapping narrows access at once, and leaving a group takes effect at the next sign-in. Other sign-in methods publish no mapping.
# Conflicts: # crates/sqlbuild-orchestration/src/authorization/main/authorize_person.rs
Explain groups_claim and group_roles, which roles and scopes a mapping may grant, and when changes take effect: groups are replaced from the ID token at every sign-in, and a mapping change applies when the server restarts. Servers sharing one control database must use identical OIDC and group-role configuration.
ISO strings with and without fractional seconds do not sort as instants, so parse both sides before comparing.
Store transactions on the Windows runner ran about 20 times slower than on Linux, so tests holding one scheduler lease across their setup were fenced regardless of test thread count.
Four contracts hold one 30-second lease across many store transactions. On a slow runner the lease expired before the call under test and the contract failed Fenced. A background renewal, as a running scheduler does, keeps the epoch; LEASE_TTL_SECONDS is unchanged.
Turning it off made the store contracts faster but did not stop leases expiring; the contracts now renew their leases instead.
# Conflicts: # tests/integration/src/sqlbuild/compiler/compiled_project/publication/_test_types.py # tests/integration/src/sqlbuild/compiler/compiled_project/publication/helpers.py
Deduplicate retried rows on execution, phase and input instead of a stored event ID; the consumption table has not shipped. The bindings reuse the project's node relations and one set of field readers, and the recent and prune queries share their ranking subquery.
kvlonge
marked this pull request as ready for review
November 5, 2026 21:46
kvlonge
enabled auto-merge (squash)
November 5, 2026 21:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Native orchestration: a fenced scheduler, Python workers and an authenticated control API, now folded with project roles, changed-only builds and non-Linux recovery so one branch merges to main.
Changes
latestcatch-up tick.roles.tomlwith node selectors, OIDC group roles (beta).sqb build --changedand the jobchangedsetting, decided from append-only_sqlbuild_consumptionrecords of each write's inputs.sqlbuild-process-ownership: worker ownership, recovery and stop on Linux, macOS and Windows; Verify job "Scheduler recovery (macOS, Windows)".Verification
make check-ci;make test12,009 pass, 2 skipped (six signal tests run with default SIGINT handling);make test-nativefor state and orchestration; hygiene scan.